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1.  Introduction 

Under  most  circumstances,  a  visual  contour  is  perceived  when  there  is  a  relatively  abrupt  change  in  the 
irradiance  received  from  adjacent  areas  of  die  visual  field,  due.  for  example,  to  a  dilfcrence  in  brightness,  local 
surface  orientation,  or  color  |see,  for  example,  the  analytic  studies  of  intensity  profiles  in  Horn  1977,  Marr 
1976a,  llinford  1981],  There  are  conditions,  however,  in  which  a  relatively  abrupt  brightness  gradient  may  be 
perceived  in  areas  of  the  visual  field  where  die  physical  stimulation  is  in  fact  roughly  homogeneous.  Schumann 
{ 1 904)  is  usually  credited  with  discovering  such  illusory  contours,  or  "subjective  contours"  as  dicy  were  named 
by  Osgood  ( 1 953).  'flic  study  of  subjective  contours  was  popularized  by  Kanis/a  1 1 955],  w  ho  presented  many 
novel  and  compelling  illustrations.  Over  the  past  tw  enty-five  years  they  have  been  the  subject  of  what  Rock 
and  Anson  [1979]  have  termed  a  “Hurry  of  research”. 

In  spite  of  the  high  level  of  attention  that  has  been  devoted  to  subjective  contours,  there  is  no  comprehen¬ 
sive  single  theory  of  their  formation  nor  of  die  perceptual  purpose  that  dicy  serve.  Among  the  more  popular 
contenders  as  a  theory  arc  (i)  Gestalt-like  accounts  of  figural  completion  (Kanis/a  1955,  1974.  1976.  Osgood 
1953],  (ii)  various  lateral  inhibition  accounts  of  enhanced  brightness  contrast  that  pay  particular  regard  to  the 
apparent  brightness  of  subjective  contours  [Brigner  and  Gallagher  1974.  1-risby  and  Clatworthy  1975.  Day 
and  Jury  1978.  1980,  Jory  and  Pay  1979,  Kennedy  and  1  cc  1976],  (iii)  interpretations  of  subjective  contours 
as  stacked  planes  (Corcn  197?..  Coren  and  Theodor  1975],  which  in  all  the  audiors’  examples  are  parallel  to 
the  frontal  plane,  with  occlusion  signalled  by  interposition  ,  and  (iv)  a  search  for  a  cognitive  interpretation 
(Gregory  1972,  Rock  and  Anson  1979],  Some  audiors  show  a  commendable  spirit  of  compromise  and  contend 
that  dicre  is  no  single  theory,  rather  die  perception  of  subjective  contours  is  inevitably  multivariate  (llalpcrn 
1981], 

With  the  exception  of  Coren’s  stacked  plane  interpretation,  subjective  cunt  >urs  arc  usually  coiisidetcd 
a  part  of  two-dimensional  figmc  perception.  They  are  interpreted  as  the  two-dimensional  bounding  contour 
of  i  figiiic.  the  competing  iheoiies  implicitly  assuming  dilloieni  accounts  ol  lUnic  noimd  sepai  ilion.  I  he 
iv  I  itioii'lwp  between  the  peiception  of  subiectite  contouis  and  die  penepue:  ol  the  llucs-dimenMon.il 


natural  world  is  either  not  considered  or  is  left  unclear.  In  this  article  we  develop  an  alternative  account  of 
subjective  contours  that  derives  from  the  developing  computational  theory  of  vision  [see  Marr  1982,  Crimson 
1981a,  Ulltnan  1979c.  Brady  1982  for  overviews]  and  dial  places  subjective  contours  squarely  as  a  part  of 
the  perception  of  natural  three-dimensional  surfaces.  In  particular,  as  we  discuss  in  more  detail  below,  we 
propose  that  subjective  contours  mark  discontinuities  in  surfaces  interpolated  from  cues  in  an  image  [similar  to 
the  proposal  made  by  Marr,  1982],  We  do  this  by  placing  the  computation  of  subjective  contours  within  die 
context  of  die  computation  of  subjective  surfaces  by  interpolation  from  scattered  cues  in  images. 

The  processes  of  perception  evolved  to  constnict  three-dimensional  interpretations  of  die  physical  world 
from  two-dimensional  images.  Although  people  have  developed  impressive  capabilities  for  interpreting  car¬ 
toons.  text,  and  other  “unnatural"  images,  figure  perception  draws  heavily  upon  die  resources  of  natural 
perception.  So  far  as  early  visual  processing  is  concerned  (where  by  early  processing  we  include  such  per¬ 
ceptual  abilities  as  edge  finding,  stereo,  shape  from  texture,  shape  from  shading,  and  shape  from  motion), 
information  in  the  image  is  used  to  compute  surface  shapes  with  little  or  no  regard  to  semantic  or  other 
“higher-lever  attributes.  That  is.  die  cues  derived  from  the  image  are  assigned  three-dimensional  attributes 
without  regard  to  the  semantic  structure  of  die  filial  perception.  Our  interpretation  of  photographs  and  line 
drawings  such  as  those  shown  in  figure  l  attest  to  die  “misapplication"  of  the  processes  of  natural  perception 
to  two-dimensional  figures.  We  have  to  be  reminded  that  the  drawings  in  figure  1  are  in  fact  two-dimensional, 
so  stiongly  biased  is  our  perceptual  system  toward  three-dimensional  interpretations.  To  be  sure,  there  are 
figures  such  as  diosc  shown  in  figure  2  that  arc  not  interpreted  in  terms  of  three-dimensional  surfaces:  but  that 
in  no  w ,i>  implies  that  there  is  an  entirely  separate  apparatus  (or  figure  perception,  or  that  ue  <i<  not  save 
upon  three-dimensional  interpretations  wherever  possible.  Indeed,  an  alternative  statement  of  the  legend  io 
figure  2  is  lh.it  it  illusliates  degenerate  cases  of  surface  pciccption.  in  which  the  thieo  duncnuon.il  siniai.es 
dcpn  led  aie  p.nalle!  lo  the  image  plane.  1 1  us  inlerpietaiion  is  at  the  heart  of  (.  ou  us  aeiotiiu  of  wifim  me 
coniouis.  Inn  l< >i  us  il  is  meu.1',  a  simple  ease  of  a  mine  rcneial  phenomenon. 
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Figure  1.  Two-dimensional  figures  given  three-dimensional  interpretations,  (a)  Two  intersecting 
planes,  (b)  A  cylinder. 

To  form  a  three-dimensional  interpretation,  it  is  necessary  to  deploy  information  not  given  explicitly  in 
the  image.  For  example,  although  a  wide  variety  of  radically  differing  surfaces  could  give  rise  to  the  line 
drawings  illustrated  in  figure  1,  the  surface  shown  in  figure  la  is  perceived  as  planar,  while  that  in  figure  Ih  is 
cylindrical.  The  discovery  and  utilization  of  prior  assumptions  that  prov  ide  the  additional  constraint  needed 
to  achieve  this  perception  is  a  key  part  of  the  computational  theory  of  vision.  Importantly,  such  constraints 
can  he  of  wide  generality.  For  example.  Bin  ford  [1981]  has  cataloged  a  number  of  constraints  of  the  form  "if 
two  lines  meet  in  an  image  at  a  corner,  then  the  corresponding  space  curves  meet  in  space."  Other  constraints 
include  constraints  on  stereo  vision  [Marr  and  I’oggto.  1979).  constraints  on  the  merpietation  of  occluding 
contours  jMarr.  1977),  constraints  on  surface  interpolation  jGrimson  lM81b|.  coir sti. tints  on  motion  interpreta¬ 
tion  |Ullman.  1979,  i.  c).  constraints  on  interpretation  of  surface  cun  touts  (Stevens.  l9Sk|.  and  Kan. tile's  (TiSl) 
discussion  of  skew  svmuk  tiy  and  parallelism. 
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Figure  1  Two-dimensional  figures  given  two-dimensional  interpretations. 

The  computational  theory  of  vision,  pioneered  by  Vtarr  (1976a.  1976b,  197b.  1982]  is  outlinec  in  section 
3.  It  first  asks  of  a  module  of  the  visual  system  what  is  its  purpose,  or  equivalent!),  what  information  it  makes 
explicit.  To  date,  considerable  effort  has  been  devoted  in  the  field  of  computational  vision  to  understanding 
stereo  (Marr  and  Poggio  1979.  Grimson  1981c,  Maylicw  and  1'risby  1981.  Moravcc  1980).  motion  (Ullman 
1979a.  l.onguet-Higgins  and  Pra/dny.  1981.  Horn  and  Schunck.  1981),  shape  from  shading  |Horn  1975.  1977. 
1981),  shape  from  texture  (Kcndcr  198(1,  Stevens  1981b)  and  shape  from  contour  (Stevens  1981c.  W;tkin  1981] 
All  of  these  modules  make  explicit  information  about  the  layout  of  visible  surfaces.  We  propose  tha:  subjective 
contours , vise  from  ihc  processes  of  natural  surface  perception,  ami  in  particular  mat  subjective  ani’ours  nark 
surface  ,h\i  ontinwtin. 

We  introduce  a  recent  theory  ofsurl.uo  interpolation  (Grimson  1981b.  Hiadv  and  Horn  1981  in  soot  ion 
3.  In  our  ac  mail  of  •airf.i-  e  mh’ipolulion.  the  liist  dunes  to  he  disvavcieJ  aie  di  -  oiiiiiuulir'  ill  a  evomu  ilh 
l"iui  p  a!  of  il.c  I  it  mi  ml.  i  rv  ol  a  ■  i-.it>lc  Mill. i  c.  \  *le  dial  die  bound. u  \  Jos  no  )i.i\ v-  |o  I-.  ,  iM.ijiht,  ii  i 


(lie  subsequent  completion  of  the  bounding  contour  that  gives  rise  to  subjective  contours.  An  interpolation 
process  is  then  applied  to  construct  a  surface  that  is  consistent  with  values  hypothesized  for  certain  points  in 
the  image.  In  general,  many  surfaces  are  consistent  with  a  given  set  of  boundary  values  and  so  additional 
constraint  is  required.  Two  constraints  are  proposed  by  the  theory,  f  irst,  Grimson  [1981b]  observes  that  “no 
information  is  information”:  since  rapid  changes  in  surface  topography  usually  give  rise  to  visible  edges  and 
visible  edges  provide  boundary  values  to  die  interpolation  process,  die  absence  of  boundary  values  means  that 
the  underlying  surface  topography  does  not  change  too  rapidly  between  discontinuities,  t  his  idea  is  made 
precise  in  the  theory  using  die  calculus  of  variations  to  find  die  surface  that  is  consistent  with  the  given  data 
and  minimizes  an  appropriate  performance  index.  Second,  there  is  a  technical  requirement  on  die  application 
of  the  calculus  of  variations  that  turns  out  to  have  an  agreeable  perceptual  interpretation:  the  surface  is 
interpolated  using  a  rotationallj  symmetric  operator  dial  minimizes  die  quadratic  variation  of  die  surface.  The 
computation  by  which  this  is  effected  can  be  implemented  by  a  local  parallel  network  of  relatively  simple 
computing  elements  [Grimson  1981a], 

Notice  that  die  interpolation  process  may  not  be  totally  constrained.  The  computation  of  surface  dis¬ 
continuities  precedes  surface  interpolation,  which  in  turn  precedes  the  computation  of  depth.  The  interpola¬ 
tion  process  determines  the  tilt  and  slant  at  all  points  on  visible  surfaces.  The  result  may  be  underdctcr- 
mined,  as  in  the  Mach  illusion.  The  computation  of  dcpdi  follows  die  interpolation  process.  It  too  may 
be  undcrconstrained.  The  Necker  reversal  is  a  familiar  example  of  this  phenomenon.  LTic  possibility  for 
undcrcomtraint  in  botli  depth  and  surface  orientation  finesses  die  arguments  of  Kennedy  [197ba.  page  110) 
and  Rock  and  Anson  |1979,  page  M»6|,  both  of  whom  assume  that  depth  is  computed  directly  and  without 
ambiguity. 

Given  dial  many  difference  cues  can  give  surface  information  and  tli.it  this  in  format  ion  is  intcipolated 
to  form  an  explicit  three-dimensional  surface  inlcrprct.il ion.  this  suggests  that  it  should  he  possible  to 
deinonsliate  subjective  surfaces.  v\  iih  associated  suh|aii\c  contours  at  the  discontinuities,  due  to  each  of  the 
possible  s  uf, ice  cues.  We  do  dus  in  section  1  Some  prior  evidence  lor  tins  can  he  found  in  C  oon  |19'n|. 
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In  section  3  we  outline  the  computational  theory  of  vision,  paying  particular  regard  to  surface  interpolation, 
which  underlies  the  computation  of  subjective  surfaces.  In  section  4  we  make  an  initial  attempt  to  construct  a 
catalog  of  three-dimensional  constraints  that  arc  hypothesized  from  certain  image  fragments,  namely  isolated 
points,  line  endings,  and  more  complex  figures.  Finally,  in  section  S  we  consider  the  relationship  between  edge 
detection  and  subjective  contours,  since  variations  on  enhanced  brightness  have  been  offered  as  evidence  in 
support  of  an  early  processing  theory  of  subjective  contours. 

2.  Ihe  three-dimensionality  of  subjective  contours 

The  previous  section  sketched  our  approach  to  figure  perception.  If  a  figure  has  a  three-dimensional 
interpretation,  our  perceptual  system  is  strongly  biased  toward  constructing  it.  We  contend  that  a  figure 
has  a  three-dimensional  interpretation  if  and  only  if  there  exists  a  subjective  contour  with  a  similar  three- 
dimensional  interpretation.  Figure  1  shows  a  set  of  line  drawings  whose  preferred  interpretation  is  three- 
dimensional.  In  each  case  there  is  a  perceived  internal  discontinuity  in  surface  orientation,  a  visible  edge. 
According  to  our  theory  of  surface  interpolation,  such  discontinuities  have  to  be  made  explicit,  albeit  in  the 
form  of  additional  subjective  contours.  Certain  line  drawings  are  interpreted  as  curv  ed  surfaces  even  though 
they  have  no  internal  edges. 

We  intend  to  demonstrate  first  the  variety  of  shapes  that  a  subjective  surface,  and  therefore  a  subjective 
contour,  can  take.  We  will  then  show  how  a  wide  variety  of  surface  cues  can  drive  the  construction  of  a 
subjective  surface  and  its  associated  subjective  contours. 

Four  such  line  drawings  provide  our  first  examples  of  subjective  surfaces.  Figure  3  shows  a  subjective 
sin  face  that  is  usually  perceived  as  a  portion  of  a  cylinder.  Notice  tli.it  the  subjective  contours  lie  on  principal 
lines  of  curvature.  It  is  in  fact  very  difficult  to  draw  a  figure  that  is  interpreted  as  a  portion  of  a  cylindrical 
sin  lai  e  in  whit  h  the  lines  draw  n  do  not  correspond  to  lines  of 'curvature.  Figure  4  is «.  siibjci  live  veoioti  of  the 
sail  illusion  Mlhoiiih  the  usual  interest  in  this  figure  steins  from  the  fact  that  ii  has  multiple  iiuc.piciations. 
lh'.  in  Iciest  li-.ie  is  the  fa.  (  tli.il  it  has  at  '  '.ft  one.  I  igiuv  .S  slinv  s  a  suhjciliv  e  <!■  >mui  and  v  tends  a  it  cvainplc 
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Figure  3.  A  portion  of  a  subjective  cylindrical  surface.  Notice  that  the  subjective  contours  lie  on 
principal  lines  of  curvature. 


Figure  5.  A  subjective  donut.  As  in  figure  3,  the  lines  that  trigger  the  three-dimensional 
interpretation  appear  to  lie  along  principal  lines  of  curvature. 


discussed  by  Kennedy  [1976a.  figure  14.  page  120].  As  in  figure  3.  die  lines  that  trigger  the  three-dimensional 
interpretation  appear  to  lie  along  principal  lines  of  curvature.  Together  with  figure  6.  which  shows  a  subjective 
so.ip  bubble,  it  demonstrates  subjective  surfaces  that  arc  doubly  curved.  Figures  3  to  6  show  that  subjective 
surfaces  need  he  neither  planar,  parallel  to  the  image  plane,  nor  singly  curved,  hut  in  lad  can  correspond  to 
any  type  of  complex  surf.  ice. 


Figure  4.  A  subjective  version  of  the  sail  illusion.  As  with  most  subjective  contours,  the  effect 
can  he  made  more  compelling  by  increasing  die  contrast  and  viewing  from  a  distance  to  narrow 
the  visual  angle. 


Figure  7.  A  subjective  hinge  formed  fiom  two  subjective  sm faces.  Suflicicnt  cues  have  been  gtun 
to  enable  Lite  perceptual  system  to  c<  instinct  the  subjective  disconiinuitv  in  smtace  oiieniaiton. 
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Figure  6.  A  subjective  soap  film.  Together  with  figure  5  this  demonstrates  Unit  subjective  surfaces 
do  not  need  to  be  planar,  parallel  to  the  image  plane,  or  even  singly  curved. 
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Figure  8.  A  subjective  staircase  that  builds  upon  the  hinge  in  figure  7  and  the  Mach  illusion 


II) 


Figure  10.  When  ihe  right  image  is  viewed  with  the  right  eye  and  the  left  image  with  the  left 
eye.  a  planar  white  triangular  surface  is  seen  floating  above  the  background.  The  triangular  surface 
is  seen  as  tilted  about  the  median  passing  (brought  the  bottom  right  vertex.  Reproduced  from 
Blomfield  [1973,  figure  2,  page  256], 


Ihe  next  two  examples  require  internal  lines  and  illustrate  die  dictum  "no  information  is  information" 
discussed  above  Figure  7  shows  a  subjective  hinge  between  two  surfaces.  Ihe  discontinuity  in  surface  orienta¬ 
tion  has  been  marked  with  enough  local  cues  to  produce  an  additional  subjective  boundary.  Figure  8  builds 
upon  figure  7  and  the  Mach  illusion  to  produce  a  subjective  staircase. 

While  most  examples  of  subjective  surfaces  in  the  literature  of  subjective  contours  are  planes  parallel  to 
the  frontal  plane,  there  arc  some  examples  of  subjectiv  e  surfaces  that  arc  not  parallel  to  the  image  plane.  For 
example.  Kennedy  (1976b,  figure  11.  page  45J  shows  a  stripe  on  a  subjective  cylnder  (see  figure  9).  Figure 
1(1  is  reproduced  from  Blomfield  (1973,  figure  2,  page  756j  and  shows  a  stereo  pair  that  fuses  to  yield  a  tilted 
surface.  Figure  1 1  is  the  subjective  Nccker  cube  due  to  Bradley  and  Betty  [1977.  figure  1.  page  254],  From  the 
standpoint  of  the  theory  advanced  here,  figure  1 1  is  interesting  for  two  reasons. 

1  list,  the  subjective  surfaces  that  It  Tin  the  vcnk.il  sides  of  the  Meeker  cube  are  tilted  relative  to  the  image 


plane.  Second,  the  example  illtisii.iics  the  point  nude  e.nliei  (lt.it  the  depth  and  mi,  •nl.iii.ui  of  m  nrei|'<>l.iied 
Mil 'jet  me  sin  laee  tail  be  linden  oioii.um  d.  vet  give  i  r*  to  the  s.une  suh|<.>  live  ton'  mis. 
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Fijjurc  9.  A  subjective  stripe  lying  on  u  cylindrical  surface.  Reproduced  from  Kennedy  [1976b. 
figure  11.  page  45]. 

In  the  previous  section  we  noted  that  according  to  the  computational  theory  of  vision,  the  purpose  of 
several  modules  of  the  visual  system  is  to  make  surface  layout  explicit.  Specifically,  stereo,  motion,  texture 
discontinuities,  and  cues  for  interposition  (sec  for  example  Hittford  |1981J)  prov  ide  evidence  for  surface  layout, 
and  can  do  so  even  in  figures.  In  the  rest  of  this  section,  we  show  that  each  of  these  cues  can  be  the  driving 
force  behind  the  construction  of  a  subjective  surface. 

figure  12  shows  line  drawings  that  arc  interpreted  as  the  superposition  of  one  figure  on  top  of  another, 
figure  12a  illustrates  wh.it  we  have  been  calling  the  degenerate  case,  in  which  the  subfiguics  are  parallel  to 
the  image  plane,  figure  !2b  illustrates  that  this  is  not  a  necessary  condition,  figure  13a  is  the  subjective 
version  of  figme  12a.  originallv  discovered  by  I’arks  |l9St)a|.  f  igure  13b  extends  I’arks'  demonstration  and  is 
the  siibju  me  vcr-ion  of  figure  121'.  I  Vmoiistraiions  of  stacked  subjective  suil.ucs  parallel  to  the  image  plane 
arc  given  In  flnvn  |I‘)/.’J  and  by  figure  14.  which  is  repiodiiicd  (join  Bradley  and  Duiuais  |I9?5.  figme  !J 


Figure  II.  The  subjective  Necker  cube  invented  In  Bradley  and  Petty  [1977.  figure  1.  page  25*11. 

Figure  15  is  due  to  K.tnis/a  (1974.  figure  33,  page  109|  and  shows  a  subjective  surface  that  is  pantile!  to  the 
image  plane  and  is  framed  by  a  tilted  surface.  In  the  same  vein,  figure  1 5b  shows  a  ring  surrounding  die  top  of 
a  planar  triangle.  More  significantly,  figure  15c  is  based  on  an  example  of  Kanis/a  and  shows  a  tilted  subjective 
surface  that  is  framed  by  one  that  is  parallel  to  the  image  plane. 

Subjective  surfaces  that  result  from  stereo  fusion  arc  particularly  interesting  from  the  standpoint  of  the 
theory  being  developed  here.  Stereo  has  been  much  studied  (for  example.  Wheatstone  1983.  1952:  Helmholtz 
1925,  Jules/  1971 1  and  has  received  considerable  attention  in  the  computational  vision  literature  (for  example. 
Marr  and  1’oggio  1979.  Crimson  1981c,  Moravec  1980,  Mathew  and  Frixby  1981,  Baker  and  Binlord  1981) 
Stereo  computes  die  angular  disparity  of  matched  edge  feature  points  of  two  images  and  (up  to  scaling)  tins 
gives  the  relative  depth  at  matched  feature  points.  To  obtain  a  complete  surface  represent. Hum.  we  can  inter¬ 
polate  between  these  known  depth  points  (in  fact,  the  apptoach  loMirlaie  intcipol  ilion  outlined  ..hove  was 
tf  >el"|s  (I  in  the  contest  of  the  output  ■  *1  steico).  l  ollowin;'  die  line  ol  te-euch  noncml  l>\  lilies  <  iu 
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Figure  12.  A  line  drawing  interpreted  as  overlapping  subfigures,  (a)  The  subfigures  are  parallel 
to  the  image  plane,  (b)  Neither  subfigurc  is  parallel  to  the  image  plane,  (c)  One  subfigure  is 
interpreted  as  parallel  to  the  image  plane,  the  other  is  not. 

would  expect  to  find  pairs  of  images,  neither  of  which  give  rise  to  monocular  subjective  contours  but  in  which 
subjective  surfaces  and  hounding  contours  are  found  when  the  images  arc  fused  stereoscopically.  This  has 
been  confirmed  on  several  occasions  in  a  variety  of  ways,  perhaps  the  most  striking  of  which  has  been  the  work 
of  Gulick  and  I  awson  [1976.  see  also  Iawson  and  Gulick,  1967,  Blomficld  1973.  Gregory  and  Harris  1974. 
Jules/,  and  F'risby  1975],  Jules/  [1971|  and  Jules/  and  l-'risby  1 1 975]  present  a  number  of  examples  of  subjective 
contours  that  result  from  stereo  fusion  using  random  dot  stereograms  and  anaglyphs. 

Gregory  [1972a,  page  52]  reports  tin  experiment  in  which  the  subfigures  of  a  subjective  contour  example 
due  to  Kanis/a  1 1 955]  are  presented  separately  to  each  eye  using  a  technique  developed  by  Witasck  |1899], 
Subjective  contours  arc  seen  following  stereo  fusing  of  the  images.  Gregory  |197?b]  measured  apparent  dis¬ 
parity  with  a  light  probe,  and  ( in  gorv  and  I  larris  (1971)  showed  lh.it  measured  di'p.uiiy  of  a  subjective  stereo 
soil. ice.  cor  responding  to  increased  stratification,  nn  leases  il  and  only  if  the  peree  ed  i  l.uity  ol  ihe  subjective 
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Figure  13.  (a)  A  subjective  surface  version  of  figure  12a.  invented  by  Parks  [1980.  figure  3.  page 
240 J.  <b)  A  subjective  version  of  figure  12b. 

surface  docs.  As  we  mentioned  earlier.  Hlomftcld  [19731  demonstrated  a  tilted  subjective  surface  when  there 
is  no  monocular  evidence  for  tilt.  Finally,  Stevens  [198 Uij  reports  a  patient  with  visual  agnosia  who  could 
not  sec  subjective  contours  when  they  were  presented  monocularly,  but  who  could  see  them  when  they  were 
presented  stcrcoscopically. 

The  familiar  cue  of  shape  from  shading  [Horn  1977,  19811  is  difficult  to  demonstrate  for  subjective  sur¬ 
faces  since  a  key  attribute  of  subjective  surfaces  is  the  relative  paucity  of  information  given  explicitly  in  the 
image  to  enable  them  to  be  constructed  by  the  perceptual  system.  It  is  necessary  to  utilize  more  indirect  kinds 
of  information,  such  as  shadow  boundaries  to  generate  subjective  surfaces.  Colon  [1972.  figure  3.  page  3n!| 
presents  a  set  of  subjectively  embossed  letters  that  stand  out  from  the  page  in  intense  side  lighting.  I  nane 
lb  shows  a  sh.ulow  on  the  sail  illusion.  Most  obsoiveis  find  that  it  makes  the  subjective  sail  piesemed  in 
ligmc  4  mine  compelling,  figure  17  illiisiraies  a  limiili.w  point  in  the  context  of  siihjeutve  suit.',  -s  I  igiuc 
I  a  is  a  •  u!>|i ; mo  figure  ili.it  i  an  be  interpreted  either  as  a  two  dimension  il  pi«-  vi  lot  oi  ..  dm  ■  d  men-  uv  il 
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Figure  14.  The  six-pointed  st;ir  configuration  discussed  b>  Bradley  and  Ouniais  [1975,  figure  3, 
page  583]. 

cone.  Figure  17b  shows  that  the  interpretation  is  disambiguated  by  apparent  shading  that  makes  the  surface 
cur\ attire  explicit.  In  this  case  tire  apparent  shadow  line  "fortunately”  lies  along  a  line  of  curvature  on  the 
cone. 

Few  subjective  surfaces  that  are  pioduced  bv  motion  arc  reported.  Motion  is  computed  at  several  succes¬ 
sive  stages  by  the  visual  system  (see  for  example  Mart  and  Lillinan  |I98()|.  I 'liman  [1979u.c]).  Smith  and  Over 
[1979|  show  that  motion  al'tei  effects  tan  he  punluccd  as  readily  by  exposure  to  subjective  contours  as  by  con¬ 
tours  defined  by  trr.idiance  disconimuities.  1  lie  computation  in  question  seems  to  be  at  the  level  of  Ullman's 
correspondence  of  primal  sketch  tokens.  K.iinadi.mdrun.  Kao.  and  Vidyasagar  (19?3|  also  demotisiralc  up- 
parent  motion  with  uhjective  contouis  at  the  same  level  of  processing.  Parks  |19K0I»)  presents  an  example 
of  motion  perception  involving  subjective  to:. anus  that  seems  to  be  .it  the  level  of  surface  perception.  When 
black  disk-,  are  moved  to  and  fro.  sublets  perceive  a  continuous  piece  of  "siring”  occluded  bv  a  subjective 
(plan  ii)si  rface. 
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Figure  15.  (a)  A  subjective  surface  that  is  parallel  to  the  image  plane  and  is  flamed  by  a  tilted 
plane.  Reproduced  from  Kanis/a  [1976.  figure  33,  page  10'))  (b)  A  ciicular  plane  surrounding  a 
subjective  surface  parallel  to  the  image  plane,  (c)  A  tilted  subjective  surface  that  is  framed  by  a 
planar  surface  parallel  to  the  image  plane.  (Compare  Kanis/a  [1974.  figure  32.  page  109).) 


Texture  gradients  have  also  been  widely  investigated  as  a  cue  to  surface  layout,  suiting  with  Gibson 
1 1950].  Corcn  (1972,  figure  7,  page  364|  presents  an  example  of  a  subjective  contour  formed  by  a  texture 
gradient.  Kennedy  [1976a,  figures  6  and  7  pages  112]  (see  figure  18)  gives  two  examples  of  subjective  inter¬ 
sections  of  tilted  planar  surfaces  (figure  18).  llic  lilt  of  die  surfaces  depicted  in  Kennedy  's  figures  is  not  com¬ 
pletely  constrained.  The  figure  illustrates  die  familiar  convex-concave  reversal.  I  hc  image  of  the  dixontimi'U 
is  constrained  however. 


In  the  previous  examples  we  have  constructed  subjective  surfaces  with  subjective  contours  as  their 
boundaries  and  internal  edges  from  the  Mach  illusion,  the  com  ex  concave  reversal  shown  in  figure  >8.  and  the 
Neekcr  cube  Ibis  suggests  that  it  might  he  possible  to  devise  subpctive  coiiolates  ol  so-called  "impo.^ible 
It.,  no  :  I'v'i1  il  po-'il'le  ti;  o  s.  local  fiapii'cnls  cl  v.lir  h  m  ike  tliiec  dun-  n  m  il  \«.ti\i\  but  wl  i<  li  d  >  n.a 


Figure  16.  A  subjective  version  of  the  sail  illusion,  made  more  compelling  by  the  addition  of 
“shadow"  information. 


Figure  17.  (a)  A  snbje.tive  tiuiuc  that  can  be  mtnptctcd  eithci  as  a  plan.it  pic  shape  or  in  three 
dimensions  as  it  cone  (h)  Apparent  shading  ili  ambieit.ites  the  cltonc  In  n:al.ntg  the  ctmatuie  of 
the  cone's  mu  hue  explicit. 
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Figure  18.  A  sujective  intersection  of  two  planar  surfaces  whose  perceived  tilt  derives  from 
the  texture  gradient.  The  intersection  is  subject  to  convex-concave  ambiguity.  Reproduced  Iron 
Kennedy  (1976a.  figure  6.  page  112]. 


make  global  three-dimensional  sense  [Penrose  and  Penrose  1958,  Draper  1981.  Huffman  1971],  Ct  rcn  (1972. 
figure  4,  page  362]  is  aware  of  impossible  figures,  and  presents  a  figure  that  appears  to  be  ■  '  dow  writing, 
but  which  is  not  globally  consistent  with  regard  to  the  position  of  the  light  source.  The  subjective  contour 
that  results  is  reported  to  be  unstable.  Similarly.  Kanis/a  [1974.  figure  34.  page  109]  presents  a  liguic  that 
appears  to  consist  of  a  planar  triangle  intersecting  a  planar  subjective  triangle  that  is  parallel  to  the  imago 
plane.  Although  this  makes  sense  locally,  it  docs  not  globally.  Figure  19  is  constructed  using  the  technique  of 
Ihadlcv  and  Pc  try  (1977,  figure  1]  and  shows  the  impossible  notched  cube  discussed  by  Draper  1 1 9801. 


In  summary,  it  appears  that  interpreting  subjective  contours  as  the  boundaries  o'"  more  oi  less  const i .mu  d 
MihjecliU’  surlu'.vs  is  capable  ol  cxpl  imiirr.  the  vast  inajonn  of  published  demouMi.iiions  .mfi  lire  mow! 
i"  : 1 1 1 1  '■  (  it  I"  '  a  v  idi- ran  e  ol  ••  i'll  if  1 1.  :s  picseniol  m  ill  is  sc.  non. 
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Figure  19.  A  subjective  contour  'brming  ;in  "impossible  figure",  the  notched  cube  of  Draper 
[1980]. 

3.  Outline  of  a  computational  theory  of  early  vision. 

Having  demonstrated  a  clear  relationship  between  subjective  contours  and  their  interpretation  as  three- 
dimensional  contours,  we  now  turn  to  the  consideration  of  a  framework  in  which  to  explain  this  connection. 
The  computational  theory  of  vision,  pioneered  by  Marr  j  1976a,  197fib,  1978.  1982],  considers  the  goal  of  the 
early  and  intermediate  human  visual  system  to  be  the  construction  of  several  representations,  each  of  which 
makes  different  visual  information  explicit.  The  primal  sketch  makes  explicit  information  about  the  changes 
in  image  irradiancc,  obtained  by  locating  the  /.cro-crossings  in  the  convolution  of  the  image  with  a  set  of 
V2(!  fillets,  where  V‘  is  the  Taplacian,  and  G  is  a  Gaussian  distribution.  Hie  2 \-l)  sketch  makes  explicit 
information  about  the  shapes  and  rellective  properties  of  surfaces  and  their  distribution  relative  to  the  viewer. 
A  itimibci  ol  di  Herein  v  isu.il  modules  compute  inhumation  about  the  2 •  I  >  skelcl  I  join  the  primal  sketili.  for 
exani|  .e.  'teico.  motion,  tc'luic.  and  siirfuv  contours.  As  well  as  local  miiI.sc  si  ipe  being  made  exploit  in 
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the  2|-l)  sketch,  contours  of  surface  discontinuities,  either  in  depth  or  in  surface  orientation,  arc  also  made 
explicit.  We  have  proposed  that  subjective  contours  arise  from  the  process  of  natural  surface  perception  at  Use 
level  of  tire  2^-1)  sketch,  and  in  particular,  that  subjective  contours  mark  these  surface  discontinuities  (similar 
to  the  proposal  of  Marr  [1982|). 

Because  all  of  the  \isual  modules  mentioned  above  use  some  form  of  die  primal  sketch  as  input,  they 
compute  explicit  surface  information  only  at  a  subset  of  the  points  in  an  image,  corresponding  to  a  subset 
of  the  zero-crossings.  This  implies  that  some  form  of  surface  interpolation  must  take  place,  to  account  for 
the  perception  of  complete  surfaces.  A  recent  theory  of  surface  interpolation  |Grimson,  19S1H.  Brady  and 
Horn.  1981]  has  proposed  that  the  visual  system  computes  a  complete  surface  representation  by  constructing  a 
surface  which  is  consistent  w  ith  surface  values  hypothesized  for  certain  points  in  die  image,  for  exai  iple.  depth 
values  computed  by  the  stereo  system  along  the  zero-crossing  contours. 

Since  in  principle  any  one  of  an  infinite  class  of  surfaces  could  fit  through  the  known  siutt.-rcd  stereo 
data,  some  additional  constraint  is  needed  to  determine  die  correct  interpolated  surface.  Gnmsou  identified 
the  surface  consistency  constraint,  informally  known  as  no  news  is  good  news  (or  no  information  is  intbnnaron). 
which  states:  The  absence  of  zero-crossings  constrains  the  possible  surfaces 

This  can  be  seen  informally  by  the  following  argument.  Suppose  one  had  a  closed  circular  zero-crossing 
contour,  along  which  the  depth  was  constant,  and  that  there  were  no  further  zero-crossings  interior  to  the 
ciiclc.  One  possible  interpolating  surface  would  be  a  flat  disk,  a  second  would  he  a  hemi-splieie  \  third 
would  be  (lie  highly  convoluted  surface  formed  by  a  radial  sine  function.  Note,  however,  dial  in  the  latter 
since  the  surface  orientation  is  undergoing  a  rapid  periodic  variation,  under  most  imaging  situation'  the  uti.n 
iiradiances  will  also  undergo  a  periodic  variation.  But  if  this  is  true,  then  there  should  be  nUlit-- -  ■.  •!  ,  •> 
zei o-i  tossings,  corresponding  to  those  variations.  Since  there  are  not.  this  surface  is  not  .m  cpt.iMo  I  h  n.  ■■ 
lav k  of  /ero  crossings  in  a  teuion  of  the  image  implies  that  there  is  no  radical  ihaivv  in  tlv  mu!  ••  i  ■  l  ,  m 
lh  it  le  'Ion  .is  well. 


More  formally,  the  surface  consistency  constraint  can  be  rephrased  using  the  calculus  of  variations  to 
state  that  the  required  surface  is  one  that  is  consistent  with  the  given  data  and  minimizes  an  appropriate  func¬ 
tional.  Simply  requiring  that  the  functional  measure  surface  consistency  is  not  suflicient  to  guarantee  a  unique 
"best"  surface  fitting  die  known  data.  The  addition  of  some  minor  constraints  on  the  form  of  the  functional, 
that  it  be  a  semi-inner  product  over  a  semi-Hilbert  space,  is  sufficient  to  guarantee  a  unique  solution  surface, 
up  to  possibly  an  element  of  die  null  space  of  the  functional.  If  a  final  requirement  of  a  rotationally  symmetric 
operator  for  minimizing  die  functional  is  added,  it  can  be  shown  drat  dicrc  is  an  infinite  family  of  possible 
functionals.  This  family  forms  a  vector  space  [Brady  and  Horn,  1981],  spanned  by  the  quadratic  variation: 

/ 1  /l  +  V%+f2yydxdy 

and  the  square  Laplacian: 

J  j  {fix  +  fyyf  dxdy. 

Finally,  by  considering  th.  size  and  structure  of  the  null  space  [Grimson,  1981b),  or  by  considering  the 
constraints  supplied  by  die  calculus  of  variations  [Brady  and  Horn,  1981).  it  can  be  shown  dial  the  optimal 
functional  is  die  quadratic  variation.  In  other  words,  die  surface  perception  problem  is  solved  by  finding  the 
surface  that  fits  through  the  known  data,  and  minimizes  die  functional  of  quadratic  variation  elsewhere.  A 
straight fot  ward  parallel,  local-support  algorithm  can  be  devised  for  computing  this  surface  [Grimson.  1981a). 

Thus,  the  surface  consistency  constraint  can  be  combined  with  a  number  of  other  simple  constraints  to 
obtain  an  algorithm  for  the  interpolated  surface  which  is  most  consistent  with  the  primal  sketch  descriptions  of 
die  image  [Grimson.  1981a.  l'tSlbJ.  The  mam  point  is  that  the  algorithm  finds  the  “most  conservative"  surface 
which  fits  the  known  data,  introducing  inflections  or  discontinuities  in  die  suiface  only  when  forced  to  b>  the 
data  (c.f.  the  notion  of  "fair  surfaces”  in  car  design  [Forrest.  1972)).  If  there  is  no  explicit  evidence  for  a  sharp 
change  in  shape,  such  as  an  inflation  in  the  surface,  then  none  is  interpolated.  On  the  othei  hand,  the  role 
of  subjective  contours  in  this  context  tv  w  becomes  clear.  Subjective  contours  aie  glued  in  the  interpolated 
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surface  representation  along  contours  at  which  the  interpolated  surface  undergoes  a  sharp  change  in  shape, 
thus  interpolating  contours  of  discontinuity  between  scattered  cues  in  the  image. 

The  crucial  point  of  the  surface  consistency  constraint  is  that  an  extended  and  continuous  surface  is  to 
be  constructed  from  a  small  set  of  cues.  There  is  a  potential  continuum  of  choices  of  finite  sets  of  cues  that 
could  be  made  explicit;  but  not  all  of  them  lead  to  the  perception  of  the  surface.  The  surface  consistency 
constraint  and  the  intcrpolativc  view  of  surface  perception  of  which  it  is  a  part,  says  that  we  can  safely  leave 
vast  amounts  of  information  out  and  still  get  the  surface.  We  need  to  make  explicit  at  least  portions  of  certain 
discontinuities:  of  tangent  to  the  bounding  contour,  of  surface  slope,  indeed  anything  that  could  lead  to  a  zero 
crossing  in  the  processing  of  an  image  of  the  surface  by  the  visual  system.  Clearly,  the  entire  discontinuity 
need  not  be  made  explicit,  else  the  entire  subjective  contour  would  have  to  be  made  explicit  at  this  level. 
The  important  factor  is  that  if  no  portion  of  a  discontinuity  is  made  explicit,  the  visual  system  corscrvatively 
assumes  "no  news  is  good  news"  and  that  nothing  is  changing. 

To  be  sure,  this  set  of  cues  is  necessary  but  not  necessarily  sufficient.  For  a  given  set  of  cues,  a  variety  of 
surfaces  can  be  interpolated.  If  there  arc  many  such,  the  subjective  suT.ice  is  at  most  faintly  pcrcei  ed.  As  die 
surface  interpolation  becomes  more  constrained,  perhaps  by  adding  more  constraints,  die  subjective  surface 
is  seen  more  persistently  and  more  clearly.  Hie  perception  of  surfaces  from  natural  images  derives  from  a 
number  of  different  types  of  cues,  and  die  interpolation  process  has  to  be  capable  accepting  such  diifcrcnt 
sorts.  Critically  the  cues  arc  interpreted  in  three  dimensions  not  two. 

As  an  example  of  this  process,  we  illustrate  in  figure  20  a  random  dot  stereogram,  and  the  interpolated 
surface  that  minimizes  the  functional  of  quadratic  variation.  Ihc  sharp  discontinuities  in  depth,  occurring 
betw  een  the  different  planes  of  the  stereogram  arc  clearly  evident. 

I  inally,  several  different  algorithms  for  finding  subjective  contours  suggest  themselves;  (1)  nterpolate 
in  two  dimensions  to  complete  a  line  drawing,  then  interpret  the  line  dta\\iin>  as  the  image  i  f  a  thue 
dimensional  'Urfuce.  The  lattei  step  could  pioceed  b>  first  interpreting  the  lines  as  tluee  (imunsi.in.il  sp.i.e 
ciUK' (• .  e  Hamm  and  I  eiieiib.ium  |1')S1|).  (2)  Inierpiel  the  loval  sues  dins,  tly  in  hive  dime umiuv  opus.  ,! 
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Figure  20.  A  random  dot  stereogram  of  a  series  of  layered  planes,  and  the  interpolated  surface 
which  minimizes  the  quadratic  variant  n  of  the  surface.  Reproduced  Irom  (irtmson  (1981a]. 

lo  the  "contour"  drawing,  then  interpolate  die  surface  between  the  given  contours.  (.1)  Proceed  from  local  cues 
to  detailed  models  of  familiar  objects,  and  use  this  knowledge  to  guide  the  filling  in  *>l'ihe  subjective  surface. 

Our  arguments  above  that  subjective  conlouis  fill  a  natural  role  as  discontinuities  of  three-dimensional 
si 1 1 late  information  would  tend  to  stipeeM  that  interpolation  of  line  drawings  in  two  dimensions  is  not  com¬ 
putational  v  sound.  I  in  liter,  the  existence  ol  subjective  contours  in  -.teieo  perception,  which  arc  not  evident 
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in  the  monocular  images  yet  are  vividly  present  in  binocular  fusion  (see  for  example  figure  20  and  (Gulick 
and  Lawson,  1976,  Lawson  and  Gulick,  1967]),  argues  against  completing  contours  in  the  image  before  in¬ 
terpreting  them  in  the  surface  representation.  On  the  other  hand,  die  ability  to  construct  complete  surface 
representations,  including  explicit  surface  discontinuities,  of  unknown  objects  or  objects  seen  for  the  first  time 
argues  against  using  local  cues  to  choose  detailed  models  of  familiar  objects  and  then  using  this  information 
to  guide  the  interpolation  process.  Thus,  in  the  contex'  of  our  computational  arguments,  die  most  likely  algo¬ 
rithm  is  one  in  which  local  cues  are  directly  interpreted  in  three  dimensions,  providing  a  "contour”  drawing  of 
surface  discontinuities,  within  which  the  surface  is  interpolated.  Note  that  while  it  may  be  possible  to  design 
an  algorithm  for  constructing  subjective  contours  that  operates  on  two-dimensional  image  construe  ts.  such  an 
algorithm  must  have  as  its  underlying  basis,  a  three-dimensional  theory. 

4.  Some  image  cues  and  their  interpretation 

In  section  1  we  argued  that  if  a  figure  has  a  three-dimensional  interpretation,  our  perceptual  system  is 
strongly  biased  toward  constructing  it.  In  particular,  die  processes  that  evolved  for  natural  three-dimensional 
early  vision  arc  “misapplied"  to  two-dimensional  figures.  By  suggesting  diat  the  pen-option  of  natural  surfaces 
is  at  the  heart  of  understanding  ihc  purpose  of  subjective  surfaces  and  contours,  such  as  those  illustrated  in 
section  2,  we  implicate  all  of  die  ideas  discussed  in  section  3,  including  edge  detection,  stereo,  structure  from 
motion,  shape  from  texture,  and  so  on.  Section  3  outlined  a  computational  approach  to  early  and  intermediate 
vision  and  described  a  theory  of  surface  perception.  By  our  account,  the  creation  of  subjective  surfaces  and 
subjective  contours  proceeds  in  three  stages:  ( 1 )  a  set  of  cues  arc  isolated  in  the  primal  sketch  computed  from 
a  figure:  (2)  die  cues  arc  interpreted  in  three  space;  (3)  a  surface  is  constructed  that  is  most  conservative  with 
re  . pul  to  meeting  Ihc  constraints  imposed  by  step  (2). 

It  follows  that  the  perception  of  subjective  mii  fates  amounts  to  no  less  th  in  umL  istaudiuc  the  appli>  aiion 
I"  iw  D  dntiv.iisioii.il  lieu  ics  ol  the  prove  >es  nl  ii. it  m  a!  p-.  if-ption  at  lea  >1  lo  the  lev  v  I  ol  vk  leiiniriin-  \  i  iH 
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surfaces.  In  particular,  wc  need  to  understand  the  way  in  which  the  elements  of  the  primal  sketch  are  inter¬ 
preted  in  three  dimensions,  and  the  constraint  that  they  impose  on  the  possibly  subjective  surfaces  of  which 
they  are  a  part.  Ihis  is  no  small  task!  It  includes,  for  example,  the  tradition  of  work  aimed  .it  understanding 
line  drawings  as  three  dimensional  scenes  (Clowes  1971.  Barrow  and  Tcncbaum  1981,  Binford  1981.  Draper 
1980,  1981,  Huffman  1971,  Mackworth  1973,  Walt/.  1975,  Sugihara  1978,  1981,  Kanade  1981J.  An  interesting 
possibility  for  future  research  is  the  use  of  subjective  contours  as  a  means  of  furthering  our  und.  islanding  of 
the  interpretation  of  line  drawings  as  three-dimensional  surfaces. 

In  tics  section  we  briefly  discuss  the  entities  that  Marr  [1976a]  proposes  are  written  into  the  primal  sketch, 
and  are  interpreted  in  the  manner  outlined  by  Binford  J1981].  The  following  subsections  discuss  dots  (or  small 
blobs),  zero-crossing  contours  (for  example,  die  bounding  contours  of  figures),  and  line  endings.  These  entities 
have  been  discussed  in  several  papers  in  the  subjective  contour  literature. 

In  general,  several  surfaces  can  be  ir.'crpolated  through  a  given  set  of  constraints.  As  further  constraint 
is  added,  some  candidate  surfaces  are  ruled  out.  It  seems  that  the  more  tightly  the  surface  is  constrained,  the 
greater  its  apparent  clarity.  Notice  that  the  apparent  brightness  may  not  increase,  even  if  die  apparent  clarity 
docs.  We  return  to  this  point  in  die  next  section.  Two  further  points  arc  in  order.  First,  the  interpolation 
scheme  that  underlies  our  account  of  surface  perception  is  capable  of  accepting  a  variety  of  cues.  This  should 
be  contrasted  with  the  schemes  devised  by  Ullman  [1976]  and  Horn  [1981],  Second,  the  interpolation  algo¬ 
rithm  docs  not  require  the  surface  to  meet  the  given  constraints  exactly.  It  incorporates  a  "penalty"  scheme 
that  allows  some  constraints  to  be  met  approximately.  This  allows  local  deviations  from  constraints  to  achieve 
the  best  global  surface.  Crimson  (1981a]  gives  details. 

Dots 


A  dot  or  small  blob  in  the  primal  sketch  marks  a  point  in  three-dimensional  space.  Although  its  position 
in  the  image  is  determined,  the  depth  from  the  viewer  is  not  (Gibson  (1979]  refers  to  such  a  constrained  point 
as  an  optie  ray).  In  general,  a  dot  may  he  in,  on.  or  neat  a  subjective  contour,  cot- ^ponding  to  a  three --.pace 
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Figure  21.  Dots  may  be  in,  on,  or  near,  a  subjective  contour. 


point  that  is  on  the  boundary  of,  contained  in.  or  outside  of,  a  subjective  surface  (see  Figure  21).  A  dot  that 
is  interpreted  as  contained  in  a  subjective  surface  appears  as  a  mark  dial  has  different  reflectance  from  its 
surrounds,  like  an  ink  blob  (see  below  also).  Points  that  are  interpreted  as  lying  outside  a  subjective  surface  arc 
not  acc  orded  significance. 

As  wc  discussed  above,  if  a  dot  is  on  or  near  a  subjective  contour,  the  contour  is  constrained  to  pass 
through  it,  and  the  subjective  surface  is  seen  more  clearly  (figure  22).  Vertices  and  the  ends  of  space  curves 
also  play  a  significant  role  in  thrcc-space  interpretations  ofline  drawings.  Fine  endings  arc  discussed  later, 
by  Grimson's  “no  news  is  good  news”  dictum,  our  perceptual  system  expects  such  points  to  be  made  explicit. 
Figures  23  and  24  show  the  effect  of  adding  points  that  mark  vertices.  Figure  24  is  particularly  interesting  as 
the  dots  disambiguate  the  shape  of  the  subjective  surface,  which  is  shown  without  the  dots  in  figure  24a  and 
appears  as  a  circle  or  a  square.  The  circle  versus  square  interpretation  is  also  alfected  by  the  visual  angle  from 
w  hich  the  figure  is  viewed. 

Mobs 

blobs  sm  h  as  that  shown  in  figure  25  appear  as  closed  zero-crossing  ioniums  in  the  primal  sketch,  and  me 
mlerpietcd  nu  l'acc  patches  nr  as  object  silhouettes  |\1urr  1  *>77.  brads  l'*7‘t|.  I  lie  chance  in  intensity  acmvs 
'■'*<  h  i  I’lrb  ins  K  en  diaraUi  ii/c<l  as  a  step  (see  for  example  Man  |  F)?(>  i).  I  leu  km  Us  and  bin Imd  |l''(l||  In 
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Figure  22.  Dots  can  provide  extra  constraint  on  a  subjective  surface,  leading  to  increased  clarity 
of  the  subjective  contour. 


Figure  23.  The  addition  of  dots  to  mark  the  vertices  of  a  subjective  figure  can  greatly  enhance 
its  clarity. 

natural  scenes,  such  intensity  changes  typically  correspond  to  hounding  contours  or  to  reflectance  boundaries. 
T  his  suggests  that  it  is  possible  to  generate  subjective  surfaces  in  which  some  blobs  are  interpreted  as  surfaces 
occluded  by  the  subjective  surface  and  some  arc  interpreted  as  reflectance  patches,  examples  of  this  sort  have 
appeared  it  the  literature,  for  example.  Kanis/a|1974,  figure  M.  page  10')]  (see  figure  2fi). 
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Figure  24.  The  addition  of  dots  to  m;irk  the  vertices  of  a  subjective  figure  can  rule  out  other 
possible  figures  by  requiring  the  contour  to  pass  through  the  dots. 


Figure  25.  A  blob  is  interpreted  as  a  surface  patch  or  as  the  bounding  silhouette  of  an  objut. 
(a)  A  blob  that  is  not  intciprclod  threc-dimensionally.  (b)  A  blob  dial  is  ii  teiptcted  as  a  tilled 
ihiec-ihniensional  surface,  (v)  A  blob  dial  is  imeipivicd  as  die  silhouette  ol  a  three  dimensional 
object  laliei  Marr  119771). 
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Figure  26.  A  "pinto"  subjective  trutngle  in  which  some  blobs  are  interpreted  as  occluded  by  the 
subjective  surface  and  others  are  interpreted  as  patches  of  surface  color. 

Recall  that  dots  constrain  the  position  of  a  subjective  surface.  Those  parts  of  a  blob  that  arc  interpreted  as 
lying  along  the  subjective  contour  impose  additional  constraint  on  its  direction  in  the  image.  More  precisely, 
they  constrain  the  image  of  (lie  tangent  to  the  space  curve  dial  forms  the  subjective  contour.  In  general,  the 
foreshortening  is  not  determined.  In  fact,  our  definition  of  a  dot  is  a  closed  zero-crossing  contour  which  is 
tot)  small  to  have  an  accurate  tangent  direction  computed  for  it.  Hie  additional  constraint  afforded  by  a  blob 
produces  clear  subjective  contours  in  cases  where  there  is  none  when  the  blobs  arc  replaced  by  dots  (sec  figure 
27). 

Ihe  ntcnsuv  change  associated  with  the  blob  is  us  important  as  the  geometric  shape  of  the  blob.  Figure 
28  shows  ihat  "empty"  blobs  arc  ,.ot  effective  for  generating  subjective  surfaces,  flic  intensity  change  across 
the  boundary  lines  of  an  "empty"  blob  is  not  step-like  |Marr  1 976a).  It  might  be  objected  that  the  concentric 
disks  usee  in  many  demonstrations  of  subjective  contouis  are  mu  solid  blobs,  but  are  "empty  "  in  some  sense. 
Discussio  i  of  this  point  is  deferred  until  the  next  section. 

Ilier  is.  however,  ionic  to  a  blob  than  a  step  ihangc  in  intensity .  Ihe  geometrical  shape  of  the  blob 
and  the  tl 'sciipiions  ( oiiipoUvl  by  the  visual  system  pi. iv  a  crucial  role.  1  he  computational  theory  ot  vision. 
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Figure  27.  (a)  The  four  dots  are  grouped  to  form  a  parallelogram,  but  there  is  no  strong  thre<- 
dimensional  percept  (b)  Replacing  the  dots  by  blobs  produces  a  tilted  subjective  surface,  (c)  \f 
two  of  the  blobs  are  replaced  by  dots,  the  subjective  surface  is  not  seen  clearly. 


(X) 

Qd  0 


Figure  28.  Imply  blobs  are  not  effective  at  generating  subjective  surfaces. 


indeed  the  crime  subieet  of  pattern  recognition,  has  not  \ct  devised  an  adequate  aa  omit  of  di.ipc  d<  ‘...opt 
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Figure  29.  The  blob  in  (b)  is  typically  described  as  a  blob  like  that  shown  in  (a)  with  a  piece 
cut  OUL 

Consider  figure  29.  For  this  article,  the  crucial  point  is  that  figure  29b  appears  to  be  "incomplete”,  to  be  a  ver¬ 
sion  of  figure  29a  from  which  a  piece  has  been  cut.  Kanisza  (1974]  has  argued  that  it  is  precisely  descriptions  of 
this  son  that  underlie  the  perception  of  subjective  contours.  An  initial  conjecture  is  that  tangent  discontinuities 
that  produce  concavities  in  otherwise  smooth  contours  typically  give  rise  to  descriptions  that  entail  missing 
parts.  The  qualifying  phrase  "otherwise  smooth"  recognizes  that  there  arc  shapes  such  as  figure  30  in  which 
tangent  discontinuities  do  not  always  signal  incisions.  Figure  31  shows  that  smooth  blobs  (of  which  the  circles 
favored  by  Kanisza  are  merely  a  special  case)  are  more  effective  for  generating  subjective  surfaces  than  those 
in  which  there  are  several  tangent  discontinuities.  Several  authors  have  pointed  out  that  extremely  angular 
blobs  arc  ineffective  at  generating  subjective  contours  (Corcn  [1972.  figure  8c.  page  365].  Kanisza  (1974.  figures 
5,  6,  page  %].  Hrigncr  and  Gallagher  (1974,  figure  2,  page  1049],  Rock  and  Anson  (1979,  figure  3,  page  667)). 
Angularity  is  neither  a  necessary  nor  sufficient  condition,  however. 

lines 


I  mes  in  an  image  give  rise  to  connected  contours  of  zero-crossings  in  the  primal  sketch.  Itinford  has 
isolated  the  following  list:  (a)  wires,  space  curves  that  arc  not  attached  to  any  so  'ace.  (b)  bounding  surl.ue 
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Figure  30.  Tangent  discontinuities  do  not  always  lead  to  descriptions  that  refer  to  pieces  being 
cut  out.  The  shape  shown  here  is  often  described  as  a  rectangle  with  a  piece  stuck  on  its  side 

t  *  r  i 

Figure  31.  Smooth  blobs  with  tangent  discontinuities  are  more  effective  lh;tn  angular  blobs  for 
generating  subjective  surfaces. 

contour,  curves  that  arc  attached  to  one  flanking  surface  and  signal  a  depth  boundary,  (c)  connect  edges, 
einvcs  that  arc  connected  to  the  two  surfaces  that  II, ink  the  curve,  (d)  curves  that  form  the  hound, tty  of  a 
surface  m  ilking,  and  (c)  curves  that  mark  the  boundary  of  a  shadow.  In  case  (c).  the  unmeet  oil  *e  may  he 
eitlici  tom''*  or  concave  with  tcspccl  to  the  viewer. 


Figure  32.  A  subjective  contour  with  a  protruding  wire. 


To  each  three-space  interpretation  of  a  line  there  is  a  subjective  contour  having  that  interpretation.  In  sec¬ 
tion  2  we  presented  examples  of  subjective  surfaces  that  included  bounding  contours  (the  usual  case),  shadow 
boundaries  (figures  16  and  17),  and  connect  edges  whose  convexity  or  concavity  was  occasionally  ambiguous 
(figures  7,  8,  11,  and  18).  The  only  interpretation  that  was  not  demonstrated  in  section  2  was  as  a  wire.  That 
deficit  is  repaired  in  figure  32. 

As  was  the  case  with  blobs  in  the  previous  subsection,  the  process  by  which  an  individual  line  in  the 
image  is  interpreted  is  not  completely  understood.  The  best  account  to  date  is  the  tradition  of  work  aimed 
at  understanding  line  drawings  of  polyhedra,  together  with  the  more  recent  work  of  Barrow  and  Tencnbaum 
[I(181J,  Ihnford  ( 198 1 J,  and  Stevens  (198 IcJ.  For  example,  we  typically  interpret  a  straight  line  as  a  straight 
space  curve,  and  a  curved  line  as  a  cur'  c  J  space  curve.  We  prefer  to  interpret  curves  lying  in  a  surface  as  lines 
of  curvature  (figures  3,  5,  and  17)  |sce,  for  example.  Stevens  l‘)81c).  It  is  difficult  to  indicate  notwero  torsion 
of  a  space  curve  by  an  isolated  (curved)  line,  so  strong  is  our  prefeience  for  planar  interpretations.  A  closed 
contour  lying  totally  within  a  subjective  contour  is  typically  interpreted  as  a  icllect  line  boundary  (see  figure 
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Figure  33.  Closed  contours  enclosed  by  a  subjective  contour  are  typically  interpreted  as  reflectance 
boundaries.  (Reproduced  from  Kanisza  (1974,  figure  36,  page  109]). 

Hitch  interpretation  of  a  line  imposes  constaint  on  die  surface  interpolation  process.  A  straight  line  either 
lies  in  a  subjective  surface  or  is  normal  to  the  surface.  Figure  34a  shows  a  subjective  tilted  plane  with  wires 
that  appear  to  be  normal  to  it,  whereas  in  figure  34b  the  wires  appear  to  be  lie  in  the  plane,  kanade  |19M] 
has  considered  the  conditions  under  which  three  straight  lines  in  an  image  can  be  the  images  of  vectors  in 
three  orthogonal  directions  in  space.  It  turns  out  that  the  more  nearly  two  of  the  image  lines  arc  parallel, 
the  more  likely  the  two  vectors  arc  to  interpreted  as  coplanar.  This  observation  underlies  Kennedy's  (1976a] 
“perpendicularity  hypothesis",  which  Parks  [1980a]  showed  to  be  inadequate.  Interestingly,  the  counter¬ 
examples  presented  by  Parks  involve  lines  that  arc  collincar  in  the  image  and  arc  perceived  as  a  single  space 
curve  occluded  by  the  subjective  surface,  'fhe  importance  of  line  orientation  is  also  shown  by  ai>  example 
of  Rock  and  Anson  1 1979.  figure  7b,  page  674],  They  report  that  the  introduction  of  the  stripes  "completely 
eliminated  the  perception  of  an  illusory  contour  ".  The  striped  texture  in  Rock  and  Anson's  figure  is  nearly 
collincar  with  one  of  the  sides  of  the  subjective  triangle.  Figure  35  shows  that  the  subjective  contoui  is  Mill 
perceived  when  it  is  not  nearly  collincar  with  the  orientation  of  the  texture. 

A  line  ending  may  be  near  a  subjective  contour,  in  which  case  the  lute  is  ty  pic itlly  interpictcd  as  occlmkd 


hi  the  subjective  surface.  (. incent  to  the  subjective  sutlacc,  or  notma!  to  it.  A  line  hose  oneni aiion  s  do-.e  to 


Figure  34.  (a)  A  lilted  subjective  planar  surface  with  wires  that  are  perceived  as  normal  to  it. 
(1))  Ihe  same  subjective  surface  with  wires  perceived  as  lying  in  it. 


Figure  35.  A  subjective  contour  despite  strong  background  texture. 

that  of  a  subjective  contour  constrains  the  subjective  contour  to  lie  along  it,  and  hence  constrains  its  tangent  in 
much  the  same  way  as  a  blob  (sec  figure  36). 

In  summary  although  the  interpretations  are  imperfectly  understood,  we  attribute  significance  to  the 
relationship  between  the  orientation  of  a  line  and  the  underlying  subjetlivc  surlacc.  lor  example,  finite  37  is 
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Figure  36.  The  lines  constrain  the  subjective  contour  to  lie  along  it. 

due  to  Parks  [1980,  figure  2,  page  362].  The  subjective  surface  in  figure  37a  is  clear  and  the  lines  arc  nterpreted 
|  as  occluded  by  it.  The  subjective  surface  in  figure  37b  is  less  dear.  There  is  no  compelling  interpret,  lion  of  die 

relationship  between  the  lines  and  the  underlying  surface. 

S.  The  role  of  apparent  brightness 

It  will  not  have  escaped  the  attention  of  readers  who  are  familiar  with  the  literature  of  subjects  c  contours 
that  this  article  lias  been  conspicuous  in  that  it  ignores  what  is  for  many  researchers  the  most  important, 
perhaps  the  determining,  characteristic  of  subjective  contours.  We  refer  to  the  apparently  enhanced  brightness 
of  the  region  enclosed  by  many  subjective  contours  relative  to  the  region  outside  the  subjective  contour. 
Broadly,  there  have  been  two  explanations  of  this  phenomenon,  and  they  have  largely  determined  the  respec¬ 
tive  authors'  approaches  to  subjective  contours  as  a  whole.  Some  authors,  notably  Corcn  [1972]  have  argued 
that  the  apparent  brightness  of  the  subjective  surface  is  a  consequence  of  the  three-dimensional  interpretation 
of  subjective  contours  as  stratified  planes  parallel  to  the  image  plane.  The  subjective  figure  is  perceptually 
nearer  the  viewer,  and  by  some  version  of  constancy  staling  therefore  appeals  blighter.  Oilier  a  itliors.  lor 
e  aripl'  I  rivhv  and  <  I  ilv niilu  |!975j.  Brignei  and  Gal!  tgiier  |I975|  Kennedy  and  I  ce  ||9  9,|  |(>,\  ,,nJT)i'. 
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Figure  37.  The  clear  subjective  con'our  in  (a)  derives  from  the  perceived  relationship  between  the 
lines  and  the  subjective  surface,  despite  the  fact  that  the  lines  in  (b)  are  more  nearly  orthogonal 
to  the  subjective  contour.  (Reproduced  from  Parks  (1980,  figure  2,  page  362}.) 

[1979],  Day  and  Jory  [1978,  1980]  prefer  an  explanation  based  on  some  version  of  lateral  inhibition.  Both 
positions  have  weak  points.  Coren  and  Theodor  [1975],  for  example,  point  out  that  there  are  comers  in  one  of 
their  images  that  ought  to  give  rise  to  an  enhanced  brightness  effect,  according  to  arguments  based  on  lateral 
inhibition  Parks  [1980b]  shows  how  confounded  the  effects  of  brightness  and  depth  are. 

In  the  previous  sections  we  have  argued  that  the  processes  of  natural  perception,  misapplied  to  two- 
dimension al  figures,  arc  at  the  heart  of  understanding  subjective  contours.  We  sketched  the  computational 
theory  of  visual  perception  pioneered  by  Marr,  paying  particular  regard  to  two  representations,  die  primal 
sketch  and  the  2 |-D  sketch.  We  noted  that  the  2  |-D  sketch  makes  explicit  information  about  visible  surfaces, 
including  surface  discontinuities,  that  is  computed,  whenever  possible,  from  stereo,  motion,  perspective,  tex¬ 
ture,  occlusion,  and  so  on.  These  processes,  in  turn,  operate  on  llie  information  made  explicit  in  the  pt final 
sketch  by  the  earlier  operations  of  edge  finding  and  grouping  [Marr  |976a|.  T  o  d  tic.  the  complete  sttueture 
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Figure  38.  Examples  of  the  convolution  of  common  triggering  elements  with  ■»  V2G  filter  at  a 
variety  of  scales.  The  values  demonstrate  enhanced  brightness,  assimilation,  and  dissimilation,  (i) 
The  result  of  processing  one  of  Kanisza's  figures  with  a  fine  filter,  (b)  The  result  of  processing 
the  same  Kanisza  figure  with  a  coarse  filler. 


of  the  primal  sketch,  and  complete  understanding  of  the  processes  of  edge  finding  and  grouping  arc  not  avail¬ 
able,  but  a  solid  start  has  been  made,  particularly  in  the  /.cro-crossing  theory  of  Marrand  Hildreth  (1980.  also 
Hildreth.  1980],  The  theory  proposes  that  lateral  inhibition  consists  of  convolving  an  image  with  operators  that 
arc  the  I  aplacian  of  a  Gaussian  distribution,  operators  that  closely  approximate  the  difference  of  Gaussians 
suggested  by  Wilson  and  Bergen  (1979]  (sec  also  Richter  and  Ullman  (1980).) 

By  our  account,  not  all  edges  are  created  equal  in  the  real  world,  and  it  is  the  job  of  early  processing  to 
discover  descriptions  of  contrast,  orientation,  finely  separated  lines,  blobs,  and  local  textures.  These  processes 
inevitably  discover  differences  in  the  two-dimensional  triggering  conditions  discussed  above  and  by  other 
authors  f  igure  38  shows  the  result  of  convolving  a  number  of  common  triggering  elements  at  a  v.uietx  ot 
w.  Tes  Ihe  icstilts  arc  preliminary,  and  will  be  the  subject  of  a  future  report,  but  they  iK.uly  nidi-  ate  -i'iiic 
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of  the  enhanced  brightness  clfects  suggested  in  the  literature,  such  as  I-rhoy  and  Clatwortliy's  claim  that  line 
endings  yield  larger  enhanced  brightness  effects  than  the  sides  of  lines,  Jory  and  Day’s  distinction  between 
assimilation  and  dissimilation  effects,  and  Smith  and  Over's  finding  that  subjective  contours  produce  tilt  and 
motion  after-effects. 


Wc  argued  above  that  the  figural  completion  problem  studied  by  Kanisza  formed  an  important  part  of 
die  perception  of  subjective  surfaces,  but  that  it  pertained  to  a  stage  of  processing  that  is  earlier  than  the 
computation  of  visible  surfaces.  Similarly,  edge  detection,  with  its  concomitant  brightness  effects  plays  an 
important  part  in  computing  the  primal  sketch.  This  is,  however,  also  an  earlier  stage  of  processing  than  the 
determination  of  visible  surfaces.  By  our  account,  both  enhanced  brightness  effects  that  stem  from  lateral 
inhibition  and  the  brightness  that  results  from  constancy  scaling  have  a  part  to  play  in  the  perception  of 
subjective  surfaces. 


7.  Conclusion 


It  was  proposed  that  subjective  contours  are  an  artifact  of  the  perception  of  natural  three-dimensional 
surfaces.  A  recent  theory  of  surface  interpolation  implies  that  “subjective  surfaces”  arc  constructed  in  the 
visual  system  by  interpolation  between  three-dimensional  values  arising  from  interpretation  of  a  variety  of 
surface  cues.  We  showed  that  subjective  surfaces  can  take  any  form,  including  singly  and  doubly  curved 
surfaces,  as  well  as  the  commonly  discussed  fronto-parallcl  planes.  In  addition,  it  is  necessary  in  the  context 
of  computational  vision  to  make  explicit  the  discontinuities,  both  in  depth  and  in  surface  orientation,  in  the 
surfaces  constructed  by  interpolation.  It  was  proposed  that  subjective  contours  form  the  boundaries  of  die 
subjects c  surfaces  due  to  these  discontinuities.  Several  novel  subjective  surfaces  and  subjective  contours  were 
demonstrated  to  support  this  position.  The  role  played  by  figure  completion  and  enhanced  brightness  contrast 
in  the  determination  of  subjective  surfaces  was  discussed. 
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